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[57] ABSTRACT 

A digital image processing system including a micro- 
processor, random access memory, storage memory, 
input means, display means, and suitable logic to pro- 
vide existing digital image processing operations in a 
conventional graphical user interface, such as a win- 
dows environment, which allows the user to implement 
image processing operations on a selected portion of a 
movie without utilizing a primitive special program- 
ming language. 
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sired scenes in a movie can be linked with correspond- 

DESKTOP DIGITA L VID EO PROCESSING ing digital video image clips. 

SYSTEM It is yet another object of the present invention to 

provide a digital video image processing system includ- 
TECHNICAL FIELD 5 ing a display which organizes and exhibits each video 

This invention relates generally to digital video clip in a two-dimensional array, providing a Storyboard 
image processing and, more particularly, to a system Vl f* ° f * e ^ keorporated mto a movic - 
including an improved user interface for performing " y<*^ 0 toer object of the present invention to 
digital video image processing operations in real time ln l^f^t^X^ Tf pTOC ?T s system 

on a general purpose d^^compnter. 10 ^T* i^^^^i on *.? r more 

r * selected clips from a plurality of dips being edited in a 

BACKGROUND ART frame-by-frame display. 

r». , . . . , It is an object of the present invention to provide a 

Digital image processing has been m use for many ^ proc ^ systemZvSg an fatL 

ITJ a PP k0at ? 0n3 ?°ff ^ZT^ 10 lS fa « ^"ding a display whilh'pVovides ali^ofa 
machine vision. A variety of digital .mage processmg pordon oTallof thTvSeo movie being edited therein, 
techniques, such as compression, warping, mathemab- It should be noted that the words "desktop" or "per- 
cal morphology, and other image combining operations sonal" computer as used herein are intended to referto 
have been implemented in the digital domain. These any of the currently available general purpose single, 
implementations, however, are often accomplished 20 user computers which may operate independently or 
through the use of complex special purpose high speed network and may incorporate suitable software to per- . 
digital hardware and/or machine specific software that form word processmg, graphics, database, and spread- 
operates, and must be programmed, at a primitive level sheet functions. 

It is, therefore, an object of the present invention to The words "video" or "movie" as used herein are 

create a digital image processing system which utilizes a 25 intended to refer to any electronic display of digital data 

general purpose desktop computer. sequences as images (and, optionally, sound). As used 

It is another object of the present invention to pro- hfrrin, a video or movie may include one or more 

vide a video digital image processing system including "clips" each of which are subsets of the movie. Each 

an interface which allows the user to employ complex clip is itself a movie comprising a sequence of one or 

image processing techniques to create and modify video 3° more frames of images (and, optionally, sound), 

movies in a conventional desktop user environment nmri nQTicp nu twcnt™™ 

It is yet another object of the present invention to DISCLOSURE OF INVENTION 

provide a digital video processing system which auto- carrying out the above and other objects, the digi- 

matically records the processing operations selected to ^ in^S 6 processing system of the present invention 

perform a particular video edit, so that variations in the 35 includes a microprocessor, random access memory, 

editing process can be accomplished by recalling the storage memory, such as magnetic disk, input means, 

original editing operations and reprocessing only those display means, and suitable logic to provide existing 

portions of the editing operations necessary to achieve a processing operations in a conventional 

different desired result graphical user interface which allows the user to imple- 

It is yet another object of the present invention to 40 ment . ^ e P ro «ssing techniques without utilizing a 

provide a digital video processing system which auto- prjjnitrye special programming language, 

matically records the processing operation selected to The image processing system of the present invention 

perform a particular video edit, so that a movie can be ^ ^° delude suitable logic for creating data flow 

edited in draft quality and, when completed, completely « &&phS re f° rdm 8 *•« "^Sf processing operations, and 

reprocessed (on the system of the present invention or 45 ^^^^T^T ^ ? e 

on a remote system) using higher resolution data. interface^ thereby aLowing for easy modification of a 

It is yet another object of the present invention to prCV,0Usly dc ^S ned xt of processmg operations and 

provide a video digital image processing system includ- Z^^il^*,^ ^ ^ 

rotation, perspective and zoo mine of a selected clip. -n.- . ., „ • ^ v . 6 _~ <-"i»- 

it ;* ,,„, „„„.u „*• tU , ~7. . The video image processing system of the present 

it is yet another object of the present invention to : ^. , , , . ~ir ., , 

. . 1 ., ,. . , J . F . , , , invention also includes an interface capable of display- 
provide a video digital image processing system includ- ^ one or more movic ^ wi^Tone movie view 

8 W ^° h ™ f0r qUlck modlficatlOT of displays some or all of the clips of a selected movie 

a selected video clip to warp an image. 5 5 playing in real time. Another movie view is formatted 

It is yet another object of the present invention to ^ a two-dimensional array into which some or all of the 

provide an interface which allows the user to create dips of a selected movie may be arranged in a Story- 

his/her own library of effects which may be recalled on board View. A third movie view is formatted as one or 

command to operate on a selected clip. more horizontal strips capable of showing each frame of 

It is yet another object of the present invention to 60 & selected clip in a Time line view to provide more 

provide a video digital image processing system includ- precise display of those clips for frame-accurate editing, 

ing an interface which allows the user to view abstract Another movie view is formatted as a conventional 

sample images which display the results of a particular word processor file editor, and allows the user to view 

operation. some or all of the clips of a movie with the associated 

It is yet another object of the present invention to 6J script in text form, 

provide a video digital image processing system which These windows, known as the Player, Storyboard, 

is integrated with conventional word processing pro- Tune, and Script Views respectively, together provide 

grams so that scripts corresponding to existing or de- a complete composition and editing environment which 
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allow the user to organize and process movie clips into FIG. 2 is a dataflow diagram for a representative 

a desired end product. movie processed i*=?ng the present invention; 

The Player View is in substantial part a frame-size FIG. 3 shows a typical multiple window display of 

window through which some or all of the clips of your the present invention in which the "Windows" menu is 

movie can be played. A Player View window includes 5 pulled down; 

standard controls to start and stop your movie, control FIG. 4 illustrates the Flayer View window; 

the sound volume and make selections. FIG. 5 illustrates the Storyboard View window; 

The Storyboard View window provides a two-di- FIG. 6 illustrates the Time View window; 

mensional view of a selected movie including each of FIG. 7 illustrates the Script View window; 

the clips similar to the way a spreadsheet provides a 10 FIG. 8 illustrates a dialogue window generated dur- 

two-dimensional view of statistical data. Each ™* n of mg implementation of the Pan-Zoom-Rotate command; 

the Storyboard array contains a movie clip. Each row ' FIG. 9 illustrates a dialogue window including a 

of the array represents a scene of the movie. This non- begin and end frame preview of a Pan-Zoom-Rotate 

linear organization of movie dips provides a powerful operation on a selected clip; 

way to edit movies without resorting to the more de- 15 ^G. 10 illustrates a processing window for the Pan- 
tailed display of a frame-by-frame view. Zoom-Rotate command; 

The Time View provides a precise frame-by-frame P** 3 - 11 illustrates a clip preview window; 

display of one or more clips of a movie for frame-accu- nG - 12 illustrates a save window generated during 

rate editing. The Tune View also contains one or more implementation of the Pan-Zoom-Rotate command; 

sound tracks associated with the displayed dips, for 20 nG - 13 iU^^ 1 ^ a window including an animated 

similar editing. This view also may contain a track dis- visw of a selected stored Pan-Zoom-Rotate operation; 

playing the process and inputs that created the clip. The _ FIG U illustrates another dialogue window for a 

Time View is particularly useful for specifying multiple „t? 0 ™" R , otate command; 

input operations such as transition and composite when na J 5 lHnst « t « dialogue window for the Warp 

frame accurate alignment of the input movies is re- command; 

quired. This linear view of video and audio material is FIG J \ 16 13 * scr 1 e f lBtt frabng an implementation of a 

also useful in making detailed editing decisions, because three dimensional Storyboard; and 

it displays both process formation and scene informa- "£Th1^S " ™pl«n««ation of 

tion so that the user can immediately see how process- J0 m embodiment of the Time View window, 

ing has been applied within a scene. In the preferred BEST MODE FOR CARRYING OUT THE 

embodiment, the output of any image operations are INVENTION 

automatically pasted in the Time View. In contrast, in „ • ... .„ , t . .. . 

, v ,., * ' ■ . „ . „ 7 Referring now to FIG. 1, the video image processing 

£ Z?7lt TJS 0 ^r^ 10 system of die present invention generaU? indicated^ 

the output of the selected operation, but the output is 35 30 mdudes a ^ nventiona i ^kJp computer 32 includ- 
mcorporated into the movie only upon command. ^ a microprocessor 34, random acc4 memory 36, 

The Scrrpt View window .s a conventional word stam ^ £ m ^ ffld ..nftable logic 40 to implement 
processor which allows the user to prepar^edit and ^ ^ processing operations in the visual program- 
view the script associated with the movie. The text m ^ environment « herdnafter described. The system 
the Script View window may be linked or cross- w ^ a ^ 42 ^ one or more m * de . 

referenced with assorted clips so that, by chekmg on vices such as a conventional keyboard 44 and/or mouse 
a particular cbp m the Player, Storyboard or Time 46. While the image processing system of the present 
views, trie system will search tor the point in the script invention preferably utilizes an Apple Macintosh II 
associated with the selected cup and enter the Script series, having a Motorola 680X0 (68020, 68030, and 
View at that point Selected clips may be cut and pasted 45 68040) series microprocessor, at least five megabytes of 
from the other views into the script, just as other RAM, at least 80 megabytes of disk memory, a conven- 
graphic or pictorial iriformation is inserted into word tional keyboard with numeric keypad, and conventional 
processor style sheets. When in Script View, the user mouse or roller ball, other commercially available gen- 
may double click on any clip to play that clip. cral purpose desktop computer hardware with similar 

The present invention also preferably accomplishes so color graphics capabilities may also be utilized for this 
many of the image processing techniques on selected purpose. 

clips by progressivdy altering the frames of the desig- Logic suitable for providing the image operations, 
nated dip on the basis of specified parameters selected file utilities, and the visual programming interface here- 
only for key frames. inafter described are preferably programmed in a Unix 
The present invention also provides a toolbox of jj language such as "C". Again, however, other similar 
digital image processing techniques induding video general purpose programming languages may be uti- 
editing, channel operations, filtering and 3-D trans- lized to implement the logic and interface described 
forms for movies which can be defined, combined, and herein. 

previewed in a visual programming environment Referring to FIG. 2, the system of the present inven- 

These and other objects, features, and advantages of 60 tion preferably employs a dataflow architecture which 
the present invention are readily apparent from the enables the system to capture and reproduce all the 
following detailed description of the best mode for operations (or steps) in the creation of a movie. At each 
carrying out the invention, taken in connection with the step, the system must interact with the user via the 
accompanying drawings. interface described hereinafter to identify the one or 

BRIEF DESCRIPTION OF DRAWINGS 65 m0re ^P" movies, the .key _ frame (preferably the first 

frame and last frame of a clip), control parameters re- 
FIG. 1 is a schematic of the system of the present quired for implementation of a selected operation. The 
invention; system then implements the selected operation, gener- 
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ates an output movie, and, if desired, saves that output therefore, very efficient in its use of computing re- 
movie as a new clip. As a result of the multiple input sources. 

nature of each process the dataflow is capturing more It will also be appreciated by those skilled in the art 
than just a list" of operations that is to be performed on that a movie can be edited using compressed image data 
a clip. Rather, it is capturing all of the movie edits and 5 which may also be at a lower frame rate and, once 
all of the combinations of clips. The mten^tionshrp completed, the dataflow graph defining the process can 
between the steps in the process is defined by the rela- be used to reprocess the entire movie using higher reso- 
tionship of the inputs and outputs of each of the steps. lution images as input This reprocessing could occur 
Thus, all aspects of movie making are captured in a ivang high resolution digital image data as input for the 
dataflow structure which defines the inputs, parameters 10 required clips. Alternatively, the assembly and edit 
and outputs) for each step. Since each input (clip) is a . process' can be implemented utilizing the dataflow 
movie and each output is a movie, each step in the graph for a movie on a different, remote system using 
process is unrelated in time to the other steps except as the original media as the input 

the output movies from any one particular step are The system of the present invention preferably un- 
required as input movies to any other particular step. 15 lizes the digital image formatting compression functions 
It will be appreciated that a set of operations and of QuickTime TM , a software package available from 
inputs could be defined and appear in the dataflow Apple Computer, Inc. MacStile tm , a software pack- 
graph shown in FIG. 2 as a single operation. For exam- age available from Software Originals, Inc. is also pref- 
ple, if the Warp, Pan-Zoom, and Superimpose opera- erably utilized to create the dataflow graphs utilized by 
tions shown at 236 were to be used with inputs other 20 the system. 

than Source Movie 1 and Source Movie 2, that combi- Referring now to FIG. 3, the display 42 provides a 
nation of operations might be grouped as a single opera- graphical user interface including a menu bar 50 con- 
don having two inputs fin place of Source Movie t and taming the names of each of the program menus 52-64. 
Source Movie 2) and one output As with conventional graphical user interfaces (GUI's), 

It will also be appreciated by those skilled in the art 25 the commands available under each of the menus 52-64 
that the dataflow processing implemented by the system may be displayed by keystroke command, or by posi- 
of the present invention is not limited simply to the tioning the arrow 66 over the desired menu name and 
video portion of a movie. There is no inherent data type clicking to "pull down" the available commands. For 
limitation in the dataflow structure of the system of the example, by clicking on the Windows Menu 64, the 
present invention. As used herein, a movie may repre- 30 various commands may be polled down and invoked, 
sent any sequence of dynamic media including video Again, as with conventional pull down menu interfaces, 
data, audio data, control data (e.g. key frame parame- the commands are invoked by holding the mouse button 
ters as a function of Time), MIDI music data or down and positioning the arrow 66 down the menu until 
SMPTE time code information. All of these different the desired command is highlighted, then releasing the 
media are time dependent, are part of the movie, and 35 mouse button. A desired command may also be invoked 
may be used as input in the dataflow process in the by striking the key combinations listed in the pull-down 
manner described herein. For example, control informa- menu opposite the command. Consistent with conven- 
tion may be used as input to a process to change the tional GUI's, the user may move from an active win- 
audio portion of a clip over Time. dow to another window by positioning the cursor to the 
; It should be noted that, from the movie dataflow 40 other window and "clicking" on it. Alternatively, the 
diagram, individual diagrams for each media type may user may position the cursor on an icon associated with 
be derived directly if so desired. Thus, it is possible to the desired window or action and "click" on the icon, 
generate the dataflow graph of all parts that operated in Again, as with conventional GUI's, the user may double 
some way on the video portion of a movie, or the audio • click on a file to load the data in the file or, if the file is 
portion of a movie, or the MIDI portion. 45 a program, run the program. For example, the user may 

Use of the dataflow architecture for the system of the position the cursor over the clip displayed in any of the 
present invention also provides for easy processing of a view windows and double click to play that clip, 
movie. After a movie (comprising one or more smaller The display 42 may also include one or more of the 
individual movies or clips) has been completed using movie views 70-76, from which the user can view and 
the interactive graphical user interface of the present 50 edit selected clips of a movie. It will be appreciated by 
invention, the dataflow graph corresponding to that those skilled in the art that each of the movie views 
interactive session of the type shown in FIG. 2 may be 70-76 provides a different perspective from which clips 
automatically created by the system and can be saved can be viewed, analyzed and modified. For example, the 
by the system and/or displayed to the user. If a change Player View 70 offers a viewing screen for playing or 
is desired, the user may identify the particular opera- 55 stopping an entire movie or selected clip therefrom, 
tion, revise the parameters, or one or more of the clip Referring to FIG. 4, the Player View 70 includes a 
inputs, for that operation, and re-execute that operation. movie controller SO which provides quick access to the 
Once the operation generates a revised output, this controls for the viewing screen 78. By clicking on the 
output can be automatical] y supplied wherever required sound button 82, the volume of the sound can be ad- 
as a new input and those operations utilizing the new 60 justed. By clicking on the play button 84, the movie can 
input can be similarly reprocessed. This "trickle down" be started and stopped. By clicking on the slider 86, the 
process continues until all operations whose input has ' user can move to any frame in the movie or clip being 
changed have been reprocessed. displayed in the viewing screen 78. The play reverse 88 

It will be appreciated by those skilled in the art that, and play forward 90 buttons can be activated to play the 
for any particular edit, only those operations whose 65 clip in the reverse or forward direction respectively for 
input has changed must be re-executed to generate the so long as the mouse button is held down. As with 
new, final movie By generating only the media that was conventional GUI's, the sizing button 92 can be acti- 
affected by the particular editing change, the system is, vated to resize the Player View 70. Similarly, the Player 
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View 70 can be quickly toggled between a selected 
small and large size by clicking on tie square 94 in the 
upper right comer of the window, and the window can 
be closed by clicking on the quit button 96 in the upper 
left comer of the window. s 

The Player View 70 also preferably includes a series 
of icons 98-102, each representing a movie view. For 
example, clicking on the Storyboard View icon 100 will 
open the Storyboard View window 72. 

The Flayer View 70 may also include other infonna- 10 
tion such as the filename of the current movie, the 
screen size, and the location of the current frame dis- 
plays. 

Referring now to FIG. S, the Storyboard View 72 
includes a two dimensional Storyboard array 104 which IS 
provides a grid upon which the clips 106-110 of a movie 
are arranged. In the preferred embodiment, up to five 
rows, labelled Scenes, and six columns, labelled Clips, 
may be displayed on a full-sized conventional screen. In 
addition, a conventional horizontal slider 112 and verti- 20 
cal slider 114 may be positioned to scroll and pan over 
the entire fifty scene x fifty clip grid. Each of the clips 
may be moved, viewed, and edited in the Storyboard 
View by selecting a clip (clip 110 is selected in FIG. 4, 
as denoted by the double-dark border) and implement- 25 
ing any of the system commands, as further described 
below. It will be appreciated by those skilled in the art 
that the Storyboard View 72 provides a clip-by-clip 
perspective that allows for easy rearrangement and 
processing of clips in a movie. The Storyboard View 30 
also makes efficient use of available screen space when 
working with a large number of clips. 

Storyboard View 72 is particularly useful when per- 
forming operations that require multiple clip input, such 
as the Arithmetic, Transition, Mix, Composite, and 35 
Merge commands found in the Combine Menu (de- 
scribed hereinafter). As' will be appreciated by those 
skilled in the art, in the case of multiple clip inputs, the 
Storyboard is acting as a video switcher, allowing the 
user to direct any particular set of clips to one of the 40 
multiple input operations. The power of this Storyboard 
architecture allows the user to first operate on clips 
with filters and segmentation tools, such as Slice (de- 
scribed hereinafter), then easily use the output of those 
operations as input to combining operations, such as 45 
composite, with no need to explicitly save intermediate 
result clips beyond simply dragging those clips to a 
Storyboard. 

When the movie in a Storyboard View is played, it 
will be played in scene order then clip order. Thus, SO 
referring again to FIG. 5, clip 106 will be played first, 
then clip 108, then clip 110. 

FIG. 16 illustrates an alternative, three dimensional 
Storyboard View which may be implemented is the 
system of the present invention. The Storyboard for a 55 
particular movie, such as "Art Master" may be opened 
in the manner described above. A clip 232 could then 
also be selected as described above. In this alternative . 
embodiment, a predefined command could then be acti- 
vated which would open yet another Storyboard View 60 
234 including each of the clips of another movie, which 
other movie corresponds to clip 232 in the "Art Mas- 
ter" Storyboard. 

As will be appre ci ated by those skilled in the art, the 
alternative three dimensional Storyboard illustrated in 63 
FIG. 16 accommodates an n-deep hierarchy which links 
a particular dip from one Storyboard (for example, clip 
232) to another Storyboard View (234) including each 



of the dips comprising clip 232. By clicking on one of 
the cKps in Storyboard View 234, the user could acti- 
vate yet another Storyboard which similarly includes 
all of the clips which farm the movie corresponding to 
the selected clip in Storyboard View 234. This hierar- 
chical structure could be augmented by providing a 
command which provides the dataflow graph identify- 
ing the inputs and the operation required to create any 
selected clip. In this manner, the user can access the 
input data and specific parameters and operations at any 
level in the hierarchy of the evolution of any selected 
clip or movie. 

FIG. 6 illustrates another movie view, the Time 
View window 74. The Time View 74 includes a hori- 
zontal strip 112 displaying the clip or clips of the movie 
along with up to eight additional horizontal sound 
bands displaying the audio data associated with the clips 
shown in the framestrip 112. At full screen size, the 
Time View window 74 displays eight frame displays 
116, each corresponding to one or more frames in the 
selected dips. By clicking on the time scale indicator 
US, the ratio of the number of frames per frame display 
116 can be adjusted from 1:1 to 1:12$. Thus, the Time 
View for a particularly long movie can be "condensed" 
so that each frame display 116 shows a representative 
one of a multiple number of frames in a sequence. Alter- 
natively, the frame display ratio can be set to one to one 
so that every frame in the movie may. be viewed and 
edited. 

As with the Storyboard View 72, conventional hori- 
zontal and vertical sliders 113 and 114 are provided to 
allow display of each of the horizontal frame displays 
120 and horizontal sound bands 118 regardless of the 
selected Window size. 

As with the Storyboard View 72, once the Time 
View 74 is activated, a frame or group of frames may be 
selected for processing using the system commands 
described below. It should be noted that by clicking on 
the frame ruler 124 the frame display 116 can be quickly 
toggled between its current ratio setting (shown in FIG. 
6 as 1:8) and 1:1. Thus, the single frame display can be 
quickly and easily accessed for frame accurate selec- 
tions. 

It should be noted that, in addition to the Player 
Storyboard, and Time View icons 98-102, a number of 
other icon buttons 126-134 are provided in the ribbon 
controller 74 for quick positioning and easy control of 
the marker! media. The clip-to-clip buttons 126, 127 
allow the user to position to the previous and next clip 
boundary {indicated by the clip markers 138) respec- 
tively. 

One or more frames can be selected, as indicated by 
the selection bar 136 by clicking and dragging along the 
horizontal frame display strip 116. Portions of the hori- 
zontal frame display strip 116 may also be selected by 
using standard Macintosh TM selection methods. A 
group of frames can be selected and played, in the 
Player View 70, by activating the Play in Selection 
co mm a nd under the movie menu, as hereinafter further 
described. Clip markers 138 identify the beginning and 
end points of each of the clips in the movie. Markers 138 
may be added or removed by clicking on a frame and 
executing the insert or remove dip marker command 
under the edit menu. 

Thus, it will be appreciated by those skilled in the art 
that tile Time View window 74 provides for precise 
frame-by-frame and clip-by-clip identification and edit- 
ing of a movie, with immediate access to the alternate 



04/15/2004, EAST Version: 1.4.1 



5,404,316 

9 10 

perspectives of the Player View 70, Storyboard View The program menus and their associated commands 

72 or Script View 76 by activating the appropriate icon are summarized below: 

or menu command. File — AH system documents are managed through 

It should be noted that, though each of FIGS. 3-18 the File Menu. These documents may include digital 
illustrate individual windows, these windows would S image files, audio files, or text (script) files. The digital 

appear on a display along with the menu bar 50 as image files may be compressed, and are formatted in 

shown in FIG. 2, so that all of the available commands conventional QuickTime tm , PICT or PICS formats, 

are available through a series of clicks of the mouse 46. Sound files arc kept in AIFF format 

It should also be noted that standard Macintosh TM The File Menu may include conventional file man- 
selection and basic editing methods (such as Cut/- 10 agement commands such as "New" for opening a new 

Copy/Paste) are utilized by the system of the present movie file, "Open" and "Open As" for opening existing 

invention. Thus, movies may be freely moved between files, "Close" for closing the currently active window, 

or within the Player, Storyboard and Time Views. "Save" and "Save As" for saving the current version of 

Drag and Drop is supported for Preview windows, and a movie. The File Menu also preferably includes a 
within the Storyboard View to move movies easily 15 "Print To Video" command which clears the computer 

while processing. display to provide a menu and icon-free environment 

FIG. 17 illustrates an alternative Time View Window (for example, a solid blue background) over which a 

240 which may be implemented in the system of the selected clip or movie is displayed so that the user can 

present invention. In this alternative Time View, the output the information displayed on the screen directly 
horizontal strip 112 displays a selected clip or clips of a 20 to tape. The File Menu also includes a "Preferences" 

movie in the same manner as described above in con- command which allows the user to set certain program 

nection with FIG. 6. In addition, one or more clips defaults, and a "Quit" command which allows the user 

242-246 corresponding to the input to the operation to exit the system. 

that produced the selected cHp at 112 may also be dis- Edit— The Edit Menu preferably includes commands 
played, along with an identification of the operation, at 25 which are used to move selected portions of media 

248. In this manner, the user may view the preceding between movies. Both sound and video media may be 

operation that resulted in the creation of the selected moved with these operations. Edit commands include 

clip on a frame-by-frame basis to determine whether, "Undo" which cancels the effects of the last command 

for example, alteration of the input and/or parameters selected, "Cut" which removes the media (sound, video 
for that operation, and reprocessing, is desirable. It 30 or both) of the current selection from the current movie 

should be noted that, as with the hierarchical structure and places it in the clipboard, "Copy" which places a 

of the three dimensional Storyboard illustrated in FIG. duplicate of the selected media into the clipboard, 

IS, the system of the present invention could be imple- 'Taste" which places media from the clipboard into a 

mented to allow the user to click on one of the input movie at the current location of the cursor, "Paste Re- 
clips 242-246 to obtain yet another Time View Window 35 place" which performs the same operation as "Paste", 

showing that input clip as the selected clip at 112, and except that the media is pasted over, and replaces, the 

further showing the input clip or clips and associated clip located at the current location, and "dear" which 

operation which yielded the selected clip. removes media from a movie without saving it to the 

Referring to FIG. 7, the Script View 76 links selected clipboard. It should be noted that, as used herein, the 
frames or clips from the movie with a script by activat- 40 clipboard is a conventional editing clipboard provided 

ing a commercially available word processing program, by the Apple Macintosh System, or an analogous work 

such as WordPerfect tm or Microsoft Word TM for the area which temporarily holds data that is being cut, 

Macintosh tm . The text of the script can be input or copied, or pasted during editing operations, 

edited just as any document might be prepared in using The Edit Menu also preferably includes "Select All" 

the word processing program. Thus, the top portion of 45 which selects all of the media in the current movie from 

the Script View 76 includes the menu bar 140 and asso- the first to the last frame, "Select None" which turns off 

dated programming guides such as the rule 142 from any existing selection, "Set Track Volume" which al- 

the word processing program. For those programs lows for the volume of the track to be set whenever an 

which allow importation of graphics into documents, a insertion point is set in one of the soundtracks in the 
clip 106 can be imported into the document For those SO Time View, and "Insert/Remove Clip Marker" which 

word processing programs that are Apple ® Qirickti- inserts or deletes a marker at the selected position of a 

me® aware, a clip can be played directly from the clip in the Time View. 

Script View by double clicking the mouse 46 on the Movie— The Movie Menu contains commands which 
image- define a movie, control the playback of the movie, and 
The system 30 also cross-references those portions of 55 provide other detailed information about the current 
the script which are marked (for example, Scene 1, Clip movie. Particular commands that may be implemented 
1 at 144 with the corresponding clip), so that if, for in the Movie Menu include "Crop" which creates a new 
example, the user was working in the Storyboard View movie from a selected portion of the input movie, "Re- 
window 72 and activated the clip at Scene 1, Clip 1 and size" which makes a new movie of a specified new size 
then activated the Script View window 76, the window 60 from a selected portion of the current movie, and "Re- 
would open with the script portioned in the window at compress" which compresses the selected media ac- 
Scene 1, Clip 1. Conversely, if the Script View is active cording to one or more conventional compression 
at, for example, Scene 1, Clip 2 and the Player View 70 schemes. 

is activated, the viewing screen 78 would display Scene The Movie Menus also contains "Set Duration" 

1, Clip 2. It will be appreciated by those skilled in the art 65 which changes the overall length of the selection in the 

that this integration of the scripting of the movie with current movie, "Vary Speed" which changes the dura- 

the video and audio portions, greatly aids in the efficient don of each frame in the current selection according to 

organization and assembly of the movie. a selected speed factor, thereby producing a new movie 
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that runs either faster or slower than the original, and 
"Play In Selection" which determines whether a se- 
lected portion or all of the current movie plays when- 
ever the play button in the movie controller is clicked. 

The Movie Menu also includes "Loop" which, if set, 
plays the movie repeatedly from beginning to end when 
the play button is clicked, "Loop Back And Forth" 
which, when activated, plays the current movie for- 
ward from the current frame to the end of the movie, 
then in reverse to the beginning of the movie and re- 
peats the loop continuously whenever the play button is 
clicked, and "Get Info" which provides detailed infor- 
mation about the current movie. 

Combine — The Combine Menu includes commands 
which combine or overlay selected portions of a movie 
in various ways. These commands may include "Transi- 
tion" which performs a selected one of various conven- 
tional selected transitions between two identified adja- 
cent clips, "Blend" which mixes selected portions of 
two clips together, "Chroma Key" which allows the 
user to select a color band in a clip (within a specified 
tolerance), which selected color band will be rendered 
transparent and through which a background clip will 
be seen. Combine also includes "Composite" which 
overlays (on a pixel-by-pixel basis) a foreground clip on 25 
a background clip using a third clip (called an "alpha 
channel mask" or "matte") to provide the information 
about how the first and second clips should be mixed, 
and "Mix" which performs algebraic additions of two 
selected inputs. 

The Combine Menu also includes the "Arithmetic" 
command which performs standard addition, subtrac- 
tion and multiplication operations between two selected 
clips. These operations are performed on a frame-by- 
frame basis and on a pixel-by-pixel basis within each 
frame pair. 

The Combine Menu also includes the "Logical" com- 
mand which combines movies using the logical "and", 
"or" and "exclusive or" ("XOr"). This command is 
particularly • useful for creating alpha channel masks 40 
from multiple black and white channel input movies. It 
can also be used on color movies for color modification 
and unusual image composites. 

The Combine Menu also includes a "Merge" com- 
mand which combines three channels into a single color 45 
movie. The three channels may represent either the 
RGB model or the HSV model. 

The Combine Menu also includes the "Replace" 
command which allows for selective replacement of 
any one channel in a color movie. SO 

Color — The Color Menu provides commands which 
operate independently on each of the color channels for 
each frame of the selected clip. Two color models are 
supported, the RGB model, containing Red, Green and 
Blue channels used by the Apple Macintosh ® operat- S3 
ing system and the HSV model containing Hue, Satura- 
tion and Value (or intensity) channels often used by 
artists. 

The Color Menu includes the "Extract Channel" 
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The Color Menu also preferably includes the 
"Edges" command which finds edges in a movie by 
locating changes in adjacent pixel values. These edges 
are assigned the color defined by the edge color buttons 
in the dialogue and are multiplied by the slider setting to 
control brightness. 

The Color Menu also preferably includes the "Nega- 
tive" command, which extracts one or mare selected 
channels from each frame of the clip.' 

Two other commands which are preferably included 
on the Color Menu are 'Tallef and "Custom Pallet". 
Pallet colorizes a black and white channel or remaps the 
colors in a color movie using a 256-color pallet. Because 
there are 256 allowable pixel values in a channel, rang- 
ing from 0 (black) to 255 (white), each can be assigned 
a specific color using a pallet A variety of pallets are 
provided from which a beginning and end pallet may be 
selected. 

This process creates a color movie from a single 
channel. If you specify different pallets for begin and 
End Times, the system creates a new color pallet for 
each frame of the clip. These pallets will very smoothly 
from the beginning of the clip to the end, exhibiting a 
slow color transition when the clip is played. 

The Custom Pallet command remaps the colors in a 
clip using a 256-color pallet in the same manner as the 
pallet command. However, Custom Pallet allows you to 
create the pallet interactively by specifying a single 
color for each end of the pallet The remainder of the 
pallet is created for you by interpolation. Custom Pallet 
allows for two different custom pallets, for the begin- 
ning and the end of the clip respectively. Pallets are 
then created for the intervening frames by interpolation 
so that the colors in the final clip change slowly from 
the begin pallet to the end pallet 

Filter — The Filter Menu provides access to a variety 
of commands which change (he appearance of a movie, ' 
taking a single clip as input, operating on its video por- 
tion, and outputung a new, modified clip. 

Commands accessed via the Filter Menu include 
"Pan Zoom" which changes the position of the video 
plane in two dimensions and to zoom up or away from 
a movie. 

The Filter Menu also includes the "PZR" or 'Tan- 
Zoom-Rotate" command which may be used to change 
the position of the video plane in three dimensions. In 
addition to the pan and zoom capabilities, PZR also 
rotates the plane of the image above any of the X, Y or 
Z axes. The PZR command will be discussed hereinaf- 
ter in further detail 

The Filter Menu also includes the "Warp" command 
which stretches the image in a selected cup as if they 
were on a flat rubber sheet This command is also de- 
scribed hereinafter in further detail. 

The Filter Menu also preferably includes "Slice" 
which replaces a range of values in a channel with a 
specified constant This command allows you to seg- 
ment objects from an input movie by specifying the 
range of values they contain. As will be appreciated by 
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command, which removes a specified channel from a 60 those skilled in the art, the default values of zero outside 



color clip and stores it as an 8~bit movie, which may 
then be displayed and used as a black and white clip. 

The Color Menu also includes the "Balance" com- 
mand, which provides time varying control over the 
color balance of a selected clip. 

The Color Menu also preferably includes the "Cycle" 
command which adds a constant to each pixel value in 
a selected channel of the input movie. 



range and two hundred fifty-five inside range of the 
slice values will create a binary mask of the selected 
range which may then be used as an alpha channel mask 
an other combine operations. This command is also 
65 useful for modifying the color information in a clip. 
Another command which is preferably included in 
the Filter Menu is "Threshold". Threshold clips a chan- 
nel at a specified value. All pixels below that value are 
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set to zero (black) while all pixels above it are set to 255 
(white). In the case of a color clip, each of the RGB 
color channels are operated on independently. The 
Threshold command is used to select objects of interest 
which are the brightest objects in a channel. This com- 5 
maud is a simpler version of the slice command. It can 
also be used for modification of the color space in color 
movies, providing an effect similar to a two-level "post- 
erize". The "Window" command is also preferably 
included in the Filter Menu. The Window command 10 
displays only a selected portion of the values in a chan- 
nel, thereby highlighting some features and hiding oth- 
ers. The range of selected values are reassigned to fill 
the entire range of 0-255. Thus, any value above the 
upper setting in the range is set to 255, and any value IS 
below the lower setting is set to zero. As a result, the 
selected range of values becomes the only visible por- 
tion of the movie. This command is useful for adjusting 
the visual appearance of a movie that may have been 
recorded too light or too dark. 20 

Another command that is preferably included in the 
Filter Menu is "Scale". Scale changes the brightness 
and contrast of a clip by changing the values of the 
pixels directly. Each pixel in the clip is multiplied by the 
scale factor. Thus, scale factors less than one dim the 25 
movie while those greater than one brighten it. Offset 
values may also be selected to similarly change the 
contrast of a clip. Offset values greater than zero (1 to 
255) reduce the contrast of a movie while those less than 
zero (— 1 to —255) increase the contrast The comtrina- 30 
tion of the scale and contrast controls provides a gen- 
eral mechanism for adjusting the display characteristic 
of a clip. The Scale command is also useful for modify- 
ing the values in a movie prior to using it in the Arith- 
metic command with another movie. This feature al- 35 
lows you to control which movie will dominate when 
the arithmetic operation is performed. Scale is also 
useful when operating on a grey scale channels for 
creating alpha channel masks. Controlling the bright* 
ness of the grey scale channel controls the dominance of 40 
the foreground movie when compositing. 

The Filter Menu also preferably includes "Con- 
volve" which provides a Time varying filter for se- 
lected clip. One of a variety of known filters, such as 
Sharpen, Smooth, Smooth Mare, Edge All, Edge Hori- 45 
zontal, Edge Vertical, Edge Diagonal/, Edge Diagonal, 
Emboss, or Shadow, may be selected for application on 
a frame-by-frame basis to a selected clip. In addition, 
different beginning and ending blends of the filtered and 
unffltered image may be selected. The filtering on inter- 50 
vening frames of the clip will be performed by creating 
the unique filter for each frame of the clip based upon 
the be ginnin g and end filter blends specified. 

"Custom" is also preferably included in the Filter 
Menu. Custom provides for ai custom three-by-three 55 
convolution kernel in addition to the above specified 
convolved kernels which filter a selected clip on a 
frame-by-frame basis. As with Convolve, a beginning 
and end blend of the filtered image with the original 
image can be specified. If the beginning and end blends 60 
are different, filtering on each of the intervening frames 
will be performed by creating a unique filter for each 
frame based upon the beginning and raid filters. In addi- 
tion, Convolve allows for specification of a scale 
(brightness) and offset (contrast) factors for each of the £3 
beginning and end points. Again, when these parame- 
ters are different for the begin and end frames, they are 
varied for each of the intervening frames. 
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Another command preferably located on the Filter 
Menu is "Shape". Shape performs mathematical mor- 
phology (such as erosion or dilation, or conventional 
combinations of these operations) on a selected clip 
using a preselected structuring element 

The Filter Menu also preferably includes "Posterize" 
which controls the number of discrete levels in a chan- 
nel. For a 24-bit color movie, there are 256 levels in 
each channel (0-255). For a 16-bit movie, there are 
typically 32 levels in each channel. Posterize reduces 
the number of discrete levels to between I and 1 6, creat- 
ing a paint-by-number or poster effect as the number of 
colors or levels are reduced. 

Another command preferably found on the Filter 
Menu is "Mosaic" which controls the apparent pixel 
size of each of the pixels in a clip. By specifying the 
horizontal and vertical size of the pixels in any one 
frame of the movie, a lower resolution version of the 
input movie can be created. The larger pixels in the 
output movie are formed by averaging the color values 
of all the pixels in the input movie that fall within the 
defined dimensions of the new pixel. As with other 
commands, different beginning and ending pixel tiw 
can be specified, and the system will operate indepen- 
dently on each frame of the clip to produce a smoothly 
varying pixel size for the clip. 

The "Plug Ins" command in also preferably included 
on the Filter Menu. Plug Ins provides a directory into 
which any conventional, commercially available image 
filtering operation, such as the Aldus Gallery of Ef- 
fects tm and other Adobe Photoshop tm compatible 
effects may be inserted. The system may perform a 
filtering operation on each frame of the selected clip 
using any one of the effects selected from this user cre- 
ated library. 

The Filter Menu also preferably includes the "Mo- 
tion Filter" and "Motion Shape" commands. Motion 
Filter and Motion Shape are temporal convolution and 
mathematical morphology operations respectively 
which use information from both the preceding and 
succeeding frames when processing a frame by utilizing 
a selected three dimensional structuring element, corre- 
sponding to single dimension structuring elements for 
each of Time periods n— 1, n, and n+l. 

Window— Referring again to FIG. 2, the Window 
Menu provides commands which allow the user to open 
any of the four movie windows. The "Show Player" 
command opens a window which displays the Player 
View. 

The "Show Story Board" command opens a window 
which displays the story board view. 

The "Show Time View" command opens a window 
which displays the Time View. 

The "Show Script" command opens a window which 
shows the Script View. 

The Window Menu also preferably includes a 
"Player Size" which provides for double size, normal 
size, half size, and quarter size default window sizes for 
the Player View window. 

The Window Menu also displays a list of the current 
opened movie windows, indicating which of those win- 
dows is active. 

FIGS. 8-14, along with the following description set 
forth in detail the operation and interface for the Pan- 
Zoom-Rotate command. 

Referring first to FIG. 8, after a user has selected a 
particular clip 106 from, for example, the Story board 
View 72, that clip may be modified by implementing the 
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Pan-Zoom-Rotate command as described above. Once If the operation is desired, the OK button 192 is acti- 
the command is selected, tfae Pan-Zoom-Rotate dia- vated, and the system displays a status window 193, 
logue window is displayed by the system. The Begin shown in FIG. 10, which tracks the frame-by-frame 
Time (first frame) and End Time (last frame) displays processing of the clip through the defined action. In 
152, 154 initially include cross hairs identifying the 5 addition, the status window display 194 displays each 
origin and the X and Y axes in space along with a four- frame as it is processed. 

quadrant perspective grids 156, 158 representing the As with all operations defined with Begin Time and 
orientation or size of the beginning and ending frames. End Time parameters that are non-identical, the system 
Each of the four quadrants of the perspective grids 156 interpolates the parameters for all of the intervening 
and 158 are preferably different colored lines such as, 10 frames the clip so processes. In the preferred embodi- 
for example, a green X first quadrant, a yellow X sec- meet, this interpolation is linear and continuous with 
and quadrant, a red X third quadrant, and a blue X the defined parameters being set for the first frame 
fourth quadrant It should be noted that the third (red) (Begin Time) and last frame (End Time) of a clip. How- 
quadrant of the grids 156, 158 do not appear in FIG. 8 ever, the system could also be designed to allow for 
(or FIG. 14) because red reproduces in black and white 15 specific parameter programming of additional key 
as black. By color coding the icons, the user can quickly frames, with linear interpolation of each of the interven- 
appreciate the degree of rotation and orientation of the ing frames between each of the defined key frames. In 
frame in three-space. * addition to this alternative discontinuous linear interpo- 

Referring again to FIG. 8, the first frame has been lation, other forms of interpolation may be provided 
rotated about the X axis by selecting X in the Begin 20 without departing from the spirit of the invention. 
Time rotation box 160 and either typing in the angle of It will, thus, be appreciated by those skilled in the ark- 
rotation (55 degrees) in a numerical box or clicking on that the system of die present invention provides key 
the ball of the thumbwheel 162 and rotating the ball to frame based dynamic filters by interpolating the process 
the desired position on the wheel, for example, at about parameters for the filtering operations on each of the 
163, and releasing the button on die mouse 46. The grid 15 intervening frames between the key frame, thereby 
156 in the Begin Time display 152 rotates as the thumb- creating powerful special effects. Pan-Zoom-Rotate, 
wheel ball 162 is moved about the circle (or as the de- and other Filter commands, generally have parameters 
sired number of degrees of rotation is entered in the that can be set to change the way they operate on an 
appropriate numerical box). Likewise, the grid 156 ex- image. Since movies have many frames of images, it 
pands or contracts in size as the zoom slider 164 is posi- 30 would be powerful to be able to set the parameters on a 
tioned. filter and process each image (frame) individually. 

In the example shown in FIG. 8, the zoom slider 164 However, such precise control would be tedious and 
is positioned to slightly reduce the frame to eight tenths time consuming. By providing a limited number of key 
of its original area. While the zoom slider 166 for the frames for each clip for which parameters must be es- 
End Time frame is positioned at normal size. It should 35 tabhshed, and interpolating those parameters for inter- 
be noted that, this conventional slider may be posi- vening frames, dynamic filtering is achieved without 
tioned to simultaneously and identically adjust the the tedium of frame-by-frame programming. This con- 
zoom values of the Begin Tune and End Time frames to cept of interpolation is utilized is the system wherever 
the same value by positioning the center slider 168. a continuous parameter exists that can be interpolated. 
Positioning bars 170 and 172 may be activated in a con- 40 Referring now to FIG. 11, upon completion of the 
ventional manner to position the Begin Time frame (as Pan-Zoom-Rotate process, a Pan-Zoom-Rotate pre- 
Qlustrated by the four quadrant grid 156) in the X and Y view window 194 is displayed showing the first frame 
axes, respectively. The center of the frame in the exam- (at Begin Time) of the clip 106. By activating the con- 
pie shown in the FIG. 8 has been moved to the right trols on the controller bar 196 in a manner similar to the 
(+X) and up (— Y) from the origin. Rotation of the four 45 movie controller 80 on the Player View window 70, the 
quadrant grid 158 representing the end frame of the clip processed clip 106 can be played. If desired, the clip can 
can be accomplished by manipulating the rotation con- be saved as a part of the movie by executing the appro- 
trols in the corresponding End Time rotation box 174 as priate commands, such as, for example, by clicking on 
well as by translating the frame relative to the origin the clip 106 in the preview display window 194, drag- 
using sliders 178 and 180. The rotation order box 182 50 ging the clip to the Storyboard, and pasting it in the 
may be set with the desired order of axial rotation. desired location on the Storyboard. 

Once the desired Begin Time and End Time sizes and Referring to FIGS. 9 and 12, any defined parameters 
positions are established, the user can view an animation . can be saved for future use by hitting the save button 
of the Pan-Zoom-Rotate effect by clicking on the Ani- 197 in the Pan-Zoom-Rotate window 150. The save 
mate box 184. The grid 156 in the Begin Time display 55 window 200 is then displayed, allowing the user to 
152 will then, move from its Begin Time size and posi- execute any one of the conventional file manipulation 
tion to the End Time size and position, thereby dynami- options. 

cally displaying the defined Pan-Zoom-Rotate action in It will be appreciated by those skilled in the art that 
abstract form. the save option allows for creation of a custom special 

Referring to FIG. 9, if the defined action still appears €0 effects library which can be recalled with a click of the 
desirable, the user may preview the clip by rffclring on mouse 46. As shown in FIGS. 13 and 14, by clicking the 
the Preview window 188. The system then previews the mouse 46 on the select button 202, a directory of prede- 
Begm Time and End Tune appearance of the clip by fined Pan-Zoom-Rotate parameters, including any de- 
generating the first frame image in the Begin Time fined by the user, may be accessed and utilized for any 
display window 152 and displaying the last frame image 65 selected clip. A window of predefined Pan-Zoom- 
in die End Time display window 154. If the operation is Rotate selections 204 is displayed. The user may select 
not desired, the operation can be canceled by activating from amongst defined- categories or particular transfer- 
the cancel button 190. mations. When a selection is made, a four-color, four- 
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quadrant grid 206 (the ted quadrant reproduced as 
black) provides an animated movie at 208 of the se- 
lected Pan-Zoom-Rotate action. The user can either 
select from a different category or different specific 
Pan-Zoom-Rotate transformation, cancel the window 
altogether by clicking on the cancel button at 207 or 
select the chosen transformation by clicking on OK at 
209. The predefined parameters are then loaded by the 
system in the Pan-Zoom-Rotate dialogue window ISO. 
Referring to FIG. 14, the predefined "Stretch & Slide" 
operation has been chosen. Note the Begin Time and 
End Time parameters are displayed for the selected 
action. 

Referring now to FIG. 15, the implementation of the 
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mented for other commands utilizing the above de- 
scribed interfaces. 

The command windows for other commands also 
si m ilarly provide an opportunity to preview the effect 
of a selected command on the first frame and last of a 
selected clip, as well as the use of anfmataH icons to 
demonstrate predefined effects where appropriate. 
Also, each command is typically implemented for a 
selected clip by activating an OK button in the com- 
mand window. 

It will be appreciated by those skilled in the art that 
the interface for any effect implemented by the system 
can include any one or more of the features described 
and shown in connection with FIGS. 8-14. The display 



^^n^Sf^u b oL f0 l! XiU ??,- e ' sele ? ti JS a particular 15 of these various command windows to prompt the input 
* tU ~ c * J ' " ~~' '"' ' of particular parameters for a selected command and a 

selected clip, and to preview the result for that clip 
provide a powerful and efficient movie editing tool, 
particularly when used in conjunction with the various 



clip 106 from the Storyboard View window 72, results 
in the display of the warp window dialogue 210. The 
first frame of the selected clip is displayed in the Begin 

Time window 152 and the last frame of the selected clip _ 

is displayed in the End Tune window 154. A grid 211 20 movie views provided by the "present invention. 



may be superimposed on the Begin Time window 152 
and the End Time window 154 by activating the grid 
button 212 (which toggles between "grid off' and "grid 
on". Varying grid sizes are also provided so that the 
control points can be placed directly over the area of 25 
the movie that yon wish to warp. To create a custom 
warp, the user simply activates the Custom button, then 
clicks and drags any control point on the grids 211, 213 
to the desired location. The portion of the frame under 
this control point will be stretched to this new location. 

As with all commands that have different Begin Time 
and End Time parameters, the position of each of the 
control points will be interpolated far each of the inter- 
vening frames in the clip between the Begin Time frame 
and the End Time frame to create a Time varying warp. 
As with Pan-Zoom-Rotate, the system also provides a 
set of defined warp effects which may be selected from 
the category box 214 and the effect bos 216, or alterna- 
tively by clicking on the directory button 218 which 
will prompt the system to generate a directory window 40 
(not shown) including a display of predefined warp 
effects. Once selected, the system utilizes the stored 
parameters for the Begin Time frame and the End Tune 
frame and displays the first frame of the clip in the 
Begin Time window 152 and the last frame of the clip in 45 
the End Time window 154, each altered as prescribed 
by the selected warp effect In addition, an mm a >«< 
icon 220 displays the defined effect 

As with the Pan-Zoom-Rotate command, custom 
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While the best mode for carrying out the invention 
nas been described in detail, those familiar with the art 
to which this invention relates will recognize various 
alternative designs and embodiments for practicing the 
invention as disclosed by the following claims. 
What is claimed is: 

1. A system for processing digital movies including; 
a general purpose computer having 

a microprocessor, 

random access memory operably connected to the 
microprocessor, and 
storage memory operably connected to the micro- 
processor; 

display means operably connected to the micro- 
processor, the display means including a plurality 
of view windows, each providing a display of a 
selected portion of a movie, each of the view win- 
dows having a different format, and wherein one of 
the view windows is a storyboard view wherein 
the window comprises a selectable portion of a 
two-dimensional array of sub-windows, and where 
each sub-window is capable of displaying a video 
image corresponding to a preselected portion of 
the movie; and 

input means for selecting one of the plurality of 
movie view windows to play a selected portion of 
the movie therein. 

2. The system of claim 1 wherein one of the view 
windows is a play view which comprises in substantial 



defined effects can be saved, thereby allowing the user 50 portion a generally rectangular area defining a single 



to create his/her own library for quick recall for future 
use. 

It will be appreciated by those skilled in the art that, 
although the warp effect only warps the image in two 
dimensions, three dimensional warps may be simulated 55 
by careful adjustment of the control points on the grids 
211, 213. 

Similar command windows are provided for each of 
the frame/clip editing commands supported by the 
system. It will be appreciated by those skilled in the art 60 
that, where useful, predefined effects can be provided 
and implemented for other commands. In the preferred 
embodiment, such predefined, and user definable, ef- 
fects are provided in a manner similar to that described 
above in connection with the Pan-Zoom-Rotate and 65 
Warp commands for the Transition, Pallet and Con- 
volve commands. It will be appreciated that similar 
predefined and user defined parameters can be imple- 



viewmg screen in which a selected portion of a movie 
can be played in response to commands provided 
through the input means. 

3. The system of claim 1 wherein the view windows 
include a time view which displays a selected movie in 
a plurality of sequential sub-windows, with each sub- 
window capable of displaying a selected portion of the 
movie. 

4. The system of claim 1 wherein the view windows 
include a script view comprising in substantial part an 
area defining a text processing area into which the text 
of a script corresponding to at least one portion of a 
selected movie can be displayed for editing. 

5. The system of claim 1 further including a second- 
ary storyboard view window comprising a selectable 
portion of an array of sub-windows, »-ar-h sub-window 
being capable of displaying a video image correspond- 
ing to a preselected portion of the movie; 
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logic means for correlating the video images in the 
secondary story board view window with one of 
the selected sub-windows in the storyboard view 
window, and 

second input means for activating the storyboard S 
view window, selecting a sub-window therein, and 
thereafter selecting the secondary storyboard view 
window, whereby the video images displayed in 
the secondary storyboard view window are input 
movies which, when suitably processed, yield the 10 
portion of the movie in the selected sub- window of 
the storyboard view window. 

6. The system of claim 1 wherein the preselected 
portions of the movies displayed in the sub- windows of 
the storyboard window are each clips comprising a IS 
plurality of frames of digital video images. 

7. The system of claim 6 including logic means for 
selecting any one of the clips contained in any one se- 
lected sub-window of the storyboard view and playing 
the clip within the sub-window. 20 

8. The system of claim 6 further including: 

input means for selecting at least one of the plurality 
of sub-windows; and 

first logic means for implementing selected process- 
ing operations on the selected clips in response to 25 
commands provided through the input means. 

9. The system of claim 8 wherein the processing oper- 
ation is a digital image processing operation. 

10. A system for processing digital movies including: 

a general purpose computer having: 30 
a microprocessor, 

random access memory operably connected to the 

microprocessor, 
storage memory operably connected to the micro- 
processor; and 35 
display means including a time view window which 
displays a selected portion of a movie in a plurality 
of sequential sub-windows, with each sub-window 
capable of displaying a selected portion of the 
movie, and wherein the selected portion of the 40 
movie displayed in each sub-window is a clip com- 
prising a sequence of frames of a digital video im- 
age, and further including means for defining a new 
clip by selecting a plurality of sub- windows, with 
the first frame of the new clip being the first frame 45 
of the clip in the first selected sub-window and the 
last frame of the new clip bung the last frame of the 
clip in the last selected sub-window. 

11. The system of claim 10 wherein the time view 
window further includes a two-dimensional array of SO 
sub-windows, in which the first row of sub-windows 
displays a selected portion of a movie, with each sub- 
window capable of displaying a selected sub-portion of 
the selected portion of the movie, and wherein the re- 
maining sub-windows in any column each display por- 55 
tions of movies corresponding to the inputs to image 
processing operations that produced the sub-portion of 
the movie displayed in the sub-window in the first row 

of that column. 

12. A system for processing digital movies including: 60 
a general purpose computer having: 

a microprocessor, 

random access memory operably connected to the 

microprocessor, 
storage memory operably connected to the micro- 65 

processor; 

display means including a plurality of view windows, 
each of the view windows having a different for- 



mat, wherein one of the view windows is a script 
view window comprising in substantial part an area 
defining a text processing area into which the text 
of a script corresponding to at least one portion of 
a selected movie can be displayed for editing, and 
wherein at least one other of the view windows 
provides a display of a selected portion of a movie; 

logic means for correlating the video images in one of 
the other view windows with the script displayed 
in the script view window; and 

input means for activating one of the other view 
windows, selecting a portion of a selected movie to 
be displayed therein, and thereafter selecting the 
script view window, whereby the portion of the 
script then displayed corresponds to the portion of 
the movie selected in the other view window. 

13. The system of claim 12 further including: 

a storyboard view window comprising a selectable 
portion of an array of sub-windows, and where 

. each sub-window is capable of displaying a video 
image corresponding to a preselected portion of 
the movie; 

logic means for correlating the video images in the 
sub-windows of the storyboard view with the 
script displayed in the script view, and 

input means for activating the storyboard view win- 
dow, selecting a sub-window therein, and thereaf- 
ter selecting the script view window, whereby the 
portion of the script then displayed corresponds to 
the portion of the movie selected in the selected 
sub- window of the storyboard view window. 

14. The system of claim 13 wherein the storyboard 
view window is a two-dimensional array of sub-win- 
dows with each row of sub-windows corresponding to 
a scene of the movie, and wherein the video images in 
the sub-windows of the storyboard view are correlated 
by scene with the script displayed in the script' view. 

15. A system for processing digital movies including: 
a general purpose computer having: 

a microprocessor, 

random access memory operably connected to the 
microprocessor, and 

storage memory operably connected to the micro- 
processor; and 
a display including a plurality of view windows, each 

providing a display of a movie, each of the view 

windows having a different format, 
means for selecting a first one of the plurality of 

movie view windows and selecting a portion of the 

movie displayed in the selected window, 
means for selecting a second one of the movie view 

windows including logic means for activating the 

second movie view window with the portion of the 

movie selected in the first selected movie view 

window. 

16. A system for processing digital movies including: 
a general purpose computer having: 

a microprocessor, 

random access memory operably connected to the 
microprocessor, 

storage memory operably connected to the micro- 
processor; 

display means including a plurality of view windows, 
each of the view windows having a different for- 
mat, wherein one of the view windows is a script 
view window comprising in substantial part an area 
defining a text processing area into which the text 
of a script corresponding to at least one portion of 
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a selected movie can be displayed for editing, and 
wherein at least one other of the view windows 
.provides a display of a selected portion of a se- 
lected movie; 

logic means for correlating the video images in one of 
the other view windows with the script displayed 
in the script view window; and 

input means for activating the script view window, 



22 

selecting a portion of the script to be displayed 
therein, and thereafter selecting one of the other 
view windows, whereby the portion of the movie 
then displayed corresponds to the portion of the 
script selected in the script view window. 
• » • • • 
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UNITED STATES PATENT AND TRADEMARK OFFICE 

CERTIFICATE OF CORRECTION 



PATENT NO. : 5,404,316 
DATED : April 4, 1995 

INVENTOR(S) : Klingler, et. al. 



It is certified that error appears in the above-indentrfied patent and that said Letters Patent is hereby 
corrected as shown below: 

Column 7, line 15, after "includes a " delete "two dimensional" and insert 
therefor — two-dimensional — . 

Column 8, line 34, after "selected" delete "Window" and insert therefor — 



Column 16, line 21, after "in the" delete "ark" and insert therefor — art — . 



window-- . 



Signed and Sealed this 
Twenty-first Day of November, 1995 
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BRUCE LEHMAN 



Attesting Officer 
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